Comparison of mental health symptoms before and during the covid

Objective To synthesise results of mental health outcomes in cohorts before and during the covid-19 pandemic.

Design Systematic review.

Data sources Medline, PsycINFO, CINAHL, Embase, Web of Science, China National Knowledge Infrastructure, Wanfang, medRxiv, and Open Science Framework Preprints.

Eligibility criteria for selecting studies Studies comparing general mental health, anxiety symptoms, or depression symptoms assessed from 1 January 2020 or later with outcomes collected from 1 January 2018 to 31 December 2019 in any population, and comprising ≥90% of the same participants before and during the covid-19 pandemic or using statistical methods to account for missing data. Restricted maximum likelihood random effects meta-analyses (worse covid-19 outcomes representing positive change) were performed. Risk of bias was assessed using an adapted Joanna Briggs Institute Checklist for Prevalence Studies.

Results As of 11 April 2022, 94 411 unique titles and abstracts including 137 unique studies from 134 cohorts were reviewed. Most of the studies were from high income (n=105, 77%) or upper middle income (n=28, 20%) countries. Among general population studies, no changes were found for general mental health (standardised mean difference (SMD)change 0.11, 95% confidence interval −0.00 to 0.22) or anxiety symptoms (0.05, −0.04 to 0.13), but depression symptoms worsened minimally (0.12, 0.01 to 0.24). Among women or female participants, general mental health (0.22, 0.08 to 0.35), anxiety symptoms (0.20, 0.12 to 0.29), and depression symptoms (0.22, 0.05 to 0.40) worsened by minimal to small amounts. In 27 other analyses across outcome domains among subgroups other than women or female participants, five analyses suggested that symptoms worsened by minimal or small amounts, and two suggested minimal or small improvements. No other subgroup experienced changes across all outcome domains. In three studies with data from March to April 2020 and late 2020, symptoms were unchanged from pre-covid-19 levels at both assessments or increased initially then returned to pre-covid-19 levels. Substantial heterogeneity and risk of bias were present across analyses.

Conclusions High risk of bias in many studies and substantial heterogeneity suggest caution in interpreting results. Nonetheless, most symptom change estimates for general mental health, anxiety symptoms, and depression symptoms were close to zero and not statistically significant, and significant changes were of minimal to small magnitudes. Small negative changes occurred for women or female participants in all domains. The authors will update the results of this systematic review as more evidence accrues, with study results posted online (https://www.depressd.ca/covid-19-mental-health).

Review registration PROSPERO CRD42020179703.

Concerns about covid-19 related mental health are substantial,123 but the sheer volume of low quality evidence has posed a barrier to evidence synthesis and decision making.456 Vast numbers of cross sectional studies have reported proportions of participants with scores above thresholds on easy-to-administer mental health scales as representing the “prevalence” of mental health problems, without comparisons with scores before the covid-19 pandemic.5 These scales are not, however, intended to estimate prevalence—thresholds are typically set for screening and to identify far more people than those who have a mental disorder; thus, proportions above thresholds substantially overestimate prevalence.7891011 Nonetheless, many study authors and media stories have concluded that the world’s population is experiencing a covid-19 mental health “pandemic” or “tsunami.”12

Many systematic reviews on covid-19 related mental health have synthesised results from cross sectional studies. Two previous systematic reviews compared pre-covid-19 findings with those during the pandemic.1314 One reviewed 65 studies published up to January 2021 and found a small increase in mental health symptoms in early 2020 (standardised mean difference (SMD) 0.11, 95% confidence interval 0.04 to 0.17).13 The other searched for studies up to March 2021, included 43 studies, and reported that combined depression and anxiety symptoms worsened early in the pandemic (SMD 0.39, 95% credible interval 0.03 to 0.76).14 Both reviews, however, searched a limited number of English language databases, and many relevant studies have been published since they were conducted.

We are conducting an ongoing series of living systematic reviews15 on covid-19 related mental health, including a review of studies that compared mental health during covid-19 with pre-pandemic levels in the same cohort.45 In the present study, we compared general mental health, anxiety symptoms, and depression symptoms in the general population and other groups during covid-19 with outcomes from the same cohorts before covid-19.

We registered our series of systematic reviews in PROSPERO, and our protocol is available online (https://osf.io/96csg/).16 Results are reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement.17 Supplementary material 1 describes minor amendments to the protocol.

We included studies on any population that compared eligible outcomes assessed from 1 January 2018 to 31 December 2019 when China first reported covid-19 to the World Health Organization,18 with the same outcomes collected from 1 January 2020 or later. Studies had to report data from cohorts comprising at least 90% of the same participants between pre-covid-19 and pandemic periods or to use statistical methods to account for missing data. We did not include repeated cross sectional surveys or studies with fewer than 100 participants.

Eligible outcomes included continuous scores on validated mental health symptom questionnaires, proportion of participants above a threshold on a validated symptom questionnaire, or proportion of participants meeting criteria for a mental disorder using a validated diagnostic interview. In our ongoing living systematic review, we defined outcomes broadly to include, for example, anxiety symptoms, depression symptoms, general mental health, stress, loneliness, anger, grief, and burnout. In the present report, we included only general mental health (eg, general symptoms, mental health related quality of life), anxiety symptoms, and depression symptoms because few studies reported other outcomes. Results for other outcome domains are available online (https://www.depressd.ca/covid-19-mental-health).

Using a strategy designed by an experienced health sciences librarian, we searched Medline (Ovid), PsycINFO (Ovid), CINAHL (EBSCO), Embase (Ovid), Web of Science Core Collection: Citation Indexes, China National Knowledge Infrastructure, Wanfang, medRxiv, and Open Science Framework Preprints. Because of the need for rapid evidence early in the pandemic, we did not formally peer review the search strategy; however, covid-19 terms were developed in collaboration with other librarians working on the topic and updated as covid-19 specific subject headings became available (see supplementary material 2). We initially searched from 31 December 2019 to 13 April 2020, then daily until 28 December 2020, and thereafter weekly.

Search results were uploaded into DistillerSR (Evidence Partners, Ottawa, Canada), where we identified and removed duplicate references. Two independent reviewers evaluated titles and abstracts in random order. If either reviewer deemed a study potentially eligible, a full text review was completed—also by two independent reviewers. Any discrepancies were resolved through consensus, with a third investigator consulted as necessary. An inclusion and exclusion coding guide was developed and pre-tested, and team members were trained over several sessions (see supplementary material 3).

One reviewer extracted data from each included study using a standardised form in DistillerSR, and a second reviewer validated the data using the DistillerSR Quality Control function. Reviewers extracted publication characteristics (eg, first author, publication year, journal); population characteristics, including eligibility criteria, recruitment method, number of participants, and population group (general population, older adults, young adults, children and adolescents, parents, university students, people with pre-existing medical conditions, people with pre-existing mental health conditions, medical staff, and groups defined by sex or gender in the present report, although we extracted any group for which we found data); mental health outcomes and assessment timing; and adequacy of study methods and reporting. We used World Bank classifications for country income and region.19 To assess risk of bias and adequacy of study methods and reporting (sampling frame, recruitment methods, sample size, setting and participant descriptions, participation or response rate, outcome assessment methods, standardisation of assessments, statistical analyses, follow-up rate), we adapted the Joanna Briggs Institute Checklist for Prevalence Studies20 (see supplementary material 4).

For continuous outcomes, we extracted estimates as the SMD effect size with 95% confidence intervals for change from pre-covid-19 to during the pandemic. If such data were not provided, we calculated Hedges g,21 as described elsewhere.22 Positive SMD estimates represent a worsening of mental health and negative estimates represent an improvement. For proportions, we calculated missing 95% confidence intervals using Agresti and Coull’s approximate method for binomial proportions.23 For proportion changes, we generated missing 95% confidence intervals using Newcombe’s method for differences between binomial proportions based on paired data,24 which requires the number of participants above a threshold to be known at both assessment points. If these data were not available, we assumed that 50% of cases above a threshold during pre-covid-19 assessments continued to be above the threshold during the pandemic. We confirmed that results did not differ substantively if we assumed values within a plausible range (30-70%).

Owing to pitfalls in interpreting proportions of participants who crossed a dichotomous threshold, we prioritised continuous data (see box 1). For each population group with continuous outcomes for at least two studies in a domain, we pooled SMDs through restricted maximum likelihood random effects meta-analysis. Heterogeneity was assessed using the I2 statistic. For studies with more than one continuous outcome in a domain (eg, two depression symptom measures), we pooled SMDs within the study before fitting the meta-analysis.

Changes in symptoms assessed with mental health patient reported outcome measures in covid-19 have been reported as changes in continuous scores and as the proportion of study participants above a threshold. Continuously measured symptom changes are presented in terms of standardised mean difference (SMD), which describe change in terms of within group standard deviations rather than raw change scores, which are measure specific and not easily compared across measures. To illustrate, the figure below shows the amount of change, assuming a normal distribution, for a SMD of 0.25. The hypothetical purple distribution represents pre-covid-19 scores and the orange distribution represents post-covid-19 scores, with a mean symptom increase of SMD 0.25. With a threshold located at 1 standard deviation above the pre-covid-19 mean, the proportion of participants above the threshold would change from 16% to 23%. With a threshold 2 standard deviations above the pre-covid-19 mean, the proportion would change from 2% to 4%.

When studies report an increase or decrease in the proportion of participants above a measure threshold, dichotomous thresholds used for this purpose are sometimes labelled as thresholds for clinically significant symptoms or as reflecting the presence of a condition, such as depression.7 These designations are not, however, based on evidence that a threshold represents a meaningful divide between impairment and non-impairment and do not reflect the presence of a mental disorder. Most commonly, these designations reflect a point on a measure that balances sensitivity and specificity when used for screening, which does not inform when score levels might become clinically meaningful.7891011

Thresholds on different symptom measures are often located at different places in the symptom distribution. This can lead to divergent estimates of proportions crossing a threshold, depending on the measure used, rather than because of actual differences in symptom changes. As the figure shows, the same change in symptoms in a hypothetical study sample would result in a 7% increase in participants at or above the threshold on one measure (first vertical line, 1 standard deviation above pre-covid-19 distribution mean) but an increase of only 2% on another (second vertical line, 2 standard deviations above pre-covid-19 distribution mean).

Although we have prioritised interpretation of changes in continuous score, in the supplementary material we also report proportions above thresholds, as they can be informative, such as when they are reported for two time points in the same study or as an indicator if some level of change may have occurred. We have, however, avoided interpretation of the magnitudes of proportions above thresholds.

Meta-analyses were performed in R (R version 3.6.3, Rstudio Version 1.2.5042) using the rma.uni function in the metafor package.25 Forest plots were generated using the forest.rma function in metafor. We characterised changes as minimal (SMD Fig 1

Flow of studies through review

Supplementary material 5 provides a more detailed version of study characteristics and outcomes. Supplementary table 1 shows the characteristics of the included studies. All cohort studies reported covid-19 outcome data collected in 2020 (nine studies from Asia in January or February, 125 studies from Asia and elsewhere in March or later). Only three cohort studies reported results from multiple time points in 2020, and one also reported results from 2021. Overall, 105 (77%) studies were from high income countries, 28 (20%) studies from upper middle income countries, one (1%) study from a mix of high income and upper middle income country samples, three (2%) studies from lower middle income countries, and none from low income countries. Fifty two (38%) studies were from Europe and Central Asia, 46 (34%) from East Asia and the Pacific, 28 (20%) from North America, and 11 (8%) from other regions. Study populations comprised adults in 104 (76%) studies; children or adolescents in 30 (24%) studies, including 27 that focused mostly or entirely on adolescents (10-19 years); and a mixture of children (Newcombe RG. Improved confidence intervals for the difference between binomial proportions based on paired data. Stat Med1998;17:2635-50. doi:10.1002/(SICI)1097-0258(19981130)17:223.0.CO;2-C. pmid:9839354OpenUrlCrossRefPubMedWeb of Science↵Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw2010;36:1-48. doi:10.18637/jss.v036.i03.OpenUrlCrossRefPubMed↵Cohen J. Statistical Power Analysis for the Behavioral Sciences.Routledge, 1988.↵Knudsen AKS, Stene-Larsen K, Gustavson K, et al. Prevalence of mental disorders, suicidal ideation and suicides in the general population before and during the COVID-19 pandemic in Norway: A population-based repeated cross-sectional analysis. Lancet Reg Health Eur2021;4:100071. doi:10.1016/j.lanepe.2021.100071. pmid:34557811OpenUrlCrossRefPubMed↵Pirkis J, John A, Shin S, et al. Suicide trends in the early months of the COVID-19 pandemic: an interrupted time-series analysis of preliminary data from 21 countries. Lancet Psychiatry2021;8:579-88. doi:10.1016/S2215-0366(21)00091-2. pmid:33862016OpenUrlCrossRefPubMed↵Pirkis J, Gunnell D, Shin S, et al. Suicide numbers during the first 9-15 months of the COVID-19 pandemic compared with pre-existing trends: An interrupted time series analysis in 33 countries. EClinicalMedicine2022;51:101573. doi:10.1016/j.eclinm.2022.101573. pmid:35935344OpenUrlCrossRefPubMed↵Dal Santo T, Sun Y, Wu Y, et al. Systematic review of mental health symptom changes by sex or gender in early-COVID-19 compared to pre-pandemic. Sci Rep2022;12:11417. doi:10.1038/s41598-022-14746-1. pmid:35794116OpenUrlCrossRefPubMed↵Connor J, Madhavan S, Mokashi M, et al. Health risks and outcomes that disproportionately affect women during the Covid-19 pandemic: A review. Soc Sci Med2020;266:113364. doi:10.1016/j.socscimed.2020.113364. pmid:32950924OpenUrlCrossRefPubMed↵United Nations Sustainable Development Group. Policy Brief: The Impact of COVID-19 on Women. Accessed 29 Jan 2023. https://unsdg.un.org/resources/policy-brief-impact-covid-19-women.↵Women UN. From insights to action: Gender equality in the wake of COVID-19. Accessed 29 Jan 2023. https://www.unwomen.org/en/digital-library/publications/2020/09/gender-equality-in-the-wake-of-covid-19.↵de Paz C, Muller M, Munoz Boudet AM, et al. Gender Dimensions of the COVID-19 Pandemic. World Bank Open Knowledge Repository, 16 Apr 2020. http://hdl.handle.net/10986/33622↵United States Census Bureau. Custodial Mothers and Fathers and Their Child Support: 2015. Accessed 29 Jan 2023. https://www.census.gov/library/publications/2018/demo/p60-262.html.↵Piquero AR, Jennings WG, Jemison E, Kaukinen C, Knaul FM. Domestic violence during the COVID-19 pandemic - Evidence from a systematic review and meta-analysis. J Crim Justice2021;74:101806. doi:10.1016/j.jcrimjus.2021.101806. pmid:36281275OpenUrlCrossRefPubMed↵Myran DT, Cantor N, Rhodes E, et al. Physician health care visits for mental health and substance use during the COVID-19 pandemic in Ontario, Canada. JAMA Netw Open2022;5:e2143160. doi:10.1001/jamanetworkopen.2021.43160. pmid:35061041OpenUrlCrossRefPubMed↵Saunders NR, Kurdyak P, Stukel TA, et al. Utilization of physician-based mental health care services among children and adolescents before and during the COVID-19 pandemic in Ontario, Canada. JAMA Pediatr2022;176:e216298. doi:10.1001/jamapediatrics.2021.6298. pmid:35129604OpenUrlCrossRefPubMed↵Pinker S. The media exaggerates negative news. This distortion has consequences. The Guardian. 17 Feb 2018. https://www.theguardian.com/commentisfree/2018/feb/17/steven-pinker-media-negative-news↵BBC. Psychology: Why bad news dominates the headlines. 29 Jul 2014. https://www.bbc.com/future/article/20140728-why-is-all-the-news-bad↵Marmet S, Wicki M, Gmel G, et al. The psychological impact of the COVID-19 crisis on young Swiss men participating in a cohort study. Swiss Med Wkly2021;151:w30028. doi:10.4414/SMW.2021.w30028. pmid:34596372OpenUrlCrossRefPubMed↵Lester D. Suicide rates before, during and after the world wars. Eur Psychiatry1994;9:262-4. https://psycnet.apa.org/record/1995-25489-001doi:10.1017/S092493380000362X.OpenUrlCrossRef↵Lester D. The effect of war on suicide rates. A study of France from 1826 to 1913. Eur Arch Psychiatry Clin Neurosci1993;242:248-9. doi:10.1007/BF02189971. pmid:8461352OpenUrlCrossRefPubMedWeb of Science↵Somasundaram DJ, Rajadurai S. War and suicide in northern Sri Lanka. Acta Psychiatr Scand1995;91:1-4. doi:10.1111/j.1600-0447.1995.tb09733.x. pmid:7754779OpenUrlCrossRefPubMedWeb of Science↵Aida T. Revisiting suicide rate during wartime: Evidence from the Sri Lankan civil war. PLoS One2020;15:e0240487. doi:10.1371/journal.pone.0240487. pmid:33112885OpenUrlCrossRefPubMed↵Marshall J. Political integration and the role of war in suicide. Soc Forces1981;59:771-85. doi:10.2307/2578193.OpenUrlCrossRefWeb of Science↵McCartan C, Adell T, Cameron J, et al. A scoping review of international policy responses to mental health recovery during the COVID-19 pandemic. Health Res Policy Syst2021;19:58. doi:10.1186/s12961-020-00652-3. pmid:33823855OpenUrlCrossRefPubMed↵Arendt F, Markiewitz A, Mestas M, Scherr S. COVID-19 pandemic, government responses, and public mental health: Investigating consequences through crisis hotline calls in two countries. Soc Sci Med2020;265:113532. doi:10.1016/j.socscimed.2020.113532. pmid:33223385OpenUrlCrossRefPubMed↵Ju Y, Zhang Y, Wang X, Li W, Ng RMK, Li L. China’s mental health support in response to COVID-19: progression, challenges and reflection. Global Health2020;16:102. doi:10.1186/s12992-020-00634-8. pmid:33092606OpenUrlCrossRefPubMed↵Wang V. China long avoided discussing mental health. The pandemic changed that. New York Times. 21 Dec 2020. Accessed 29 Jan 2023. https://www.nytimes.com/2020/12/21/world/asia/china-covid-mental-health.html.↵Pierce M, McManus S, Jessop C, et al. Says who? The significance of sampling in mental health surveys during COVID-19. Lancet Psychiatry2020;7:567-8. doi:10.1016/S2215-0366(20)30237-6. pmid:32502467OpenUrlCrossRefPubMed